Learning method and testing method for monitoring blind spot of vehicle, and learning device and testing device using the same

ABSTRACT

A learning method of a CNN (Convolutional Neural Network) for monitoring one or more blind spots of a monitoring vehicle is provided. The learning method includes steps of: a learning device instructing a detector to output class information and location information on a monitored vehicle in a training image; instructing a cue information extracting layer to output cue information on the monitored vehicle by using the outputted information, and instructing an FC layer to determine whether the monitored vehicle is located on the blind spots by neural-network operations with the cue information or its processed values; and learning parameters of the FC layer and parameters of the detector, by backpropagating loss values for the blind spots by referring to the determination and its corresponding GT and backpropagating loss values for the vehicle detection by referring to the class information and the location information and their corresponding GT, respectively.

FIELD OF THE INVENTION

The present invention relates to a learning method and a testing methodfor monitoring one or more blind spots of one or more vehicles, and alearning device and a testing device using the same; and moreparticularly, to the learning method of a CNN (Convolutional NeuralNetwork) for monitoring one or more blind spots of a monitoring vehicle,including steps of: (a) a learning device, if a training imagecorresponding to at least one video image taken from the monitoringvehicle is inputted, instructing a detector on the monitoring vehicle tooutput class information and location information on a monitored vehicleincluded in the training image; (b) the learning device instructing acue information extracting layer to perform one or more operations byusing the class information and the location information on themonitored vehicle, to thereby output one or more pieces of cueinformation on the monitored vehicle, and instructing an FC layer formonitoring the blind spots to perform one or more neural networkoperations by using said pieces of cue information or their processedvalues on the monitored vehicle, to thereby output a result ofdetermining whether the monitored vehicle is located on one of the blindspots of the monitoring vehicle; and (c) the learning device instructinga first loss layer to generate one or more loss values for the blindspots by referring to the result and its corresponding first GT (GroundTruth), to thereby learn one or more parameters of the FC layer formonitoring the blind spots by backpropagating the loss values for theblind spots, and instructing a second loss layer to generate one or moreloss values for vehicle detection by referring to the class informationand the location information on the monitored vehicle and theircorresponding second GT, to thereby learn one or more parameters of thedetector by backpropagating the loss values for the vehicle detection,and the testing method, the learning device, and the testing deviceusing the same.

BACKGROUND OF THE INVENTION

A monitoring vehicle has a side view mirror for each side thereof and arear-view mirror at the front center of its cabin for a good field ofview of the side and the rear needed for change of lanes by a driver.

Although the side view mirror is used for seeing each side and the rearthereof, it has a blind spot (BS) where the driver cannot see amonitored vehicle or any other objects that are very close thereto.

This has been a problem because there can be an accident with themonitored vehicle in the blind spot if the driver changes lanes withoutseeing the monitored vehicle.

To prevent such a problem, the driver sometimes put a convex mirror ontoa corner of the side view mirror, which enables the driver to see theblind spot.

However, even when the convex mirror is added onto the side view mirror,the driver must see the blind spot with his/her own eyes to change laneswhich puts further strain to the driver, and there may exist the blindspot that still cannot be seen through the convex mirror even if thedriver alters his/her head position.

To prevent this, a conventional blind spot monitoring system wassuggested that aims to prevent accidents from happening when the driverchanges lanes without noticing the monitored vehicle in the blind spot,by providing the driver with information on a detection of the monitoredvehicle, that is located in the blind spot or approaching the blindspot, through a sensor placed at the rear of the monitoring vehicle.

Especially, the conventional blind spot monitoring system using a visionsensor may detect monitored vehicles included in video information anddetermine whether the monitored vehicles are on the blind spots by usinginformation on the detected monitored vehicles.

In detail, the conventional blind spot monitoring system using thevision sensor may need some logic to determine whether the monitoredvehicles are located on the blind spots by using output signals from adetector which detects the monitored vehicles in the video information.

However, the conventional blind spot monitoring system using the visionsensor may have a problem that an appropriate logic capable ofdetermining whether the monitored vehicles are on the blind spots has tobe designed depending on the detector to be used.

In addition, the conventional blind spot monitoring system may haveanother problem of requiring lots of time to develop because theappropriate logic capable of determining whether the monitored vehiclesare on the blind spots has to be designed depending on outputcharacteristics of the designed detector.

SUMMARY OF THE INVENTION

It is an object of the present invention to solve all the aforementionedproblems.

It is another object of the present invention to provide a blind spotmonitoring system regardless of a type of a detector which detects avehicle.

It is still another object of the present invention to determine whetherthe vehicle is located on blind spots by using an output signals fromthe detector, regardless of the type of the detector.

It is still yet another object of the present invention to provide theblind spot monitoring system capable of allowing the detector to bereplaced if necessary.

It is still yet another object of the present invention to minimize timerequired to develop the blind spot monitoring system.

In accordance with one aspect of the present invention, there isprovided a learning method of a CNN (Convolutional Neural Network) formonitoring one or more blind spots of a monitoring vehicle, includingsteps of: (a) a learning device, if a training image corresponding to atleast one video image taken from the monitoring vehicle is inputted,instructing a detector on the monitoring vehicle to output classinformation and location information on a monitored vehicle included inthe training image; (b) the learning device instructing a cueinformation extracting layer to perform one or more operations by usingthe class information and the location information on the monitoredvehicle, to thereby output one or more pieces of cue information on themonitored vehicle, and instructing an FC layer for monitoring the blindspots to perform one or more neural network operations by using saidpieces of cue information or their processed values on the monitoredvehicle, to thereby output a result of determining whether the monitoredvehicle is located on one of the blind spots of the monitoring vehicle;and (c) the learning device instructing a first loss layer to generateone or more loss values for the blind spots by referring to the resultand its corresponding first GT (Ground Truth), to thereby learn one ormore parameters of the FC layer for monitoring the blind spots bybackpropagating the loss values for the blind spots, and instructing asecond loss layer to generate one or more loss values for vehicledetection by referring to the class information and the locationinformation on the monitored vehicle and their corresponding second GT,to thereby learn one or more parameters of the detector bybackpropagating the loss values for the vehicle detection.

As one example, the detector is a vehicle detector based on an R-CNN(Region-based Convolutional Neural Network) including: one or moreconvolutional layers which generate a feature map from the trainingimage, an RPN (Region Proposal Network) which generates an ROI (RegionOf Interest) of the monitored vehicle from the feature map, a poolinglayer which generates a feature vector by pooling an area, in thefeature map, corresponding to the ROI, at least one FC layer for thevehicle detection which performs at least one fully connected operationon the feature vector, to thereby generate one or more FC output values,a classification layer which outputs the class information on themonitored vehicle by referring to the FC output values, and a regressionlayer which outputs the location information on the monitored vehicle byreferring to the FC output values.

As one example, the loss values for the vehicle detection include one ormore class loss values and one or more location loss values of themonitored vehicle, wherein the learning device learns one or moreparameters of the FC layer for the vehicle detection and one or moreparameters of the convolutional layers by backpropagating the class lossvalues and the location loss values.

As one example, the learning device instructs the CNN to receive aconcatenated value acquired by using said pieces of cue information onthe monitored vehicle generated by the cue information extracting layerand the feature vector generated by the pooling layer of the detector,as an input to the FC layer for monitoring the blind spots.

As one example, the FC layer for monitoring the blind spots includes aneural network capable of outputting a result value of whether themonitored vehicle is located on one of the blind spots by a multilayerperceptron to which said pieces of cue information on the monitoredvehicle are inputted.

As one example, said pieces of cue information on the monitored vehicleinclude at least part of (i) the class information on the monitoredvehicle, (ii) the location information on the monitored vehicle, (iii)size information on the monitored vehicle corresponding to an ROI(Region Of Interest) size, (iv) aspect ratio information on themonitored vehicle, and (v) distance information between a center of themonitored vehicle and one of outer sides of the blind spots.

As one example, the cue information extracting layer determines asmaller value among either of (i) a distance of one outer boundary ofthe blind spot BS-L from the center of the monitored vehicle and (ii) adistance of the other outer boundary of the blind spot BS-R from thecenter thereof as the distance information between the center of themonitored vehicle and said one of the outer sides of the blind spots,and further outputs relative location information in addition to thedistance information in order to determine whether the center of themonitored vehicle is located inside or outside of said one of the outersides of the blind spots.

In accordance with another aspect of the present invention, there isprovided a testing method of a CNN (Convolutional Neural Network) formonitoring one or more blind spots of a monitoring vehicle, includingsteps of: (a) on condition that a learning device, (i) has instructed adetector on the monitoring vehicle to output class information fortraining and location information for training on a monitored vehicleincluded in a training image corresponding to at least one video imagetaken from the monitoring vehicle, (ii) has instructed a cue informationextracting layer to perform one or more operations by using the classinformation for training and the location information for training onthe monitored vehicle, to thereby output one or more pieces of cueinformation for training on the monitored vehicle, (iii) has instructedan FC layer for monitoring the blind spots to perform one or more neuralnetwork operations by using said pieces of cue information for trainingor their processed values, to thereby output a result for training ofdetermining whether the monitored vehicle is located on one of the blindspots, and (iv) has instructed a first loss layer to generate one ormore loss values for the blind spots by referring to the result fortraining and its corresponding first GT (Ground Truth), to thereby learnone or more parameters of the FC layer for monitoring the blind spots bybackpropagating the loss values for the blind spots, and has instructeda second loss layer to generate one or more loss values for vehicledetection by referring to the class information for training and thelocation information for training and their corresponding second GT, tothereby learn one or more parameters of the detector by backpropagatingthe loss values for the vehicle detection, a testing device instructinga detector on the monitoring vehicle to output class information fortesting and location information for testing on a monitored vehicleincluded in a test image taken from the monitoring vehicle; and (b) thetesting device instructing the cue information extracting layer toperform the operations by using the class information for testing andthe location information for testing on the monitored vehicle, tothereby output one or more pieces of cue information for testing on themonitored vehicle, and instructing the FC layer for monitoring the blindspots to perform the neural network operations by using said pieces ofcue information for testing or their processed values on the monitoredvehicle, to thereby output a result for testing of determining whetherthe monitored vehicle is located on one of the blind spots of themonitoring vehicle.

As one example, the detector is a vehicle detector based on an R-CNN(Region-based Convolutional Neural Network) including: one or moreconvolutional layers which generate a feature map for testing from thetest image, an RPN (Region Proposal Network) which generates an ROI(Region Of Interest) for testing of the monitored vehicle from thefeature map for testing, a pooling layer which generates a featurevector for testing by pooling an area, in the feature map for testing,corresponding to the ROI for testing, at least one FC layer for thevehicle detection which performs at least one fully connected operationon the feature vector for testing, to thereby generate one or more FCoutput values, a classification layer which outputs the classinformation for testing on the monitored vehicle by referring to the FCoutput values, and a regression layer which outputs the locationinformation for testing on the monitored vehicle by referring to the FCoutput values.

As one example, the testing device instructs the CNN to receive aconcatenated value for testing acquired by using said pieces of cueinformation for testing on the monitored vehicle generated by the cueinformation extracting layer and the feature vector for testinggenerated by the pooling layer of the detector, as an input to the FClayer for monitoring the blind spots.

As one example, the FC layer for monitoring the blind spots includes aneural network capable of outputting a result value for testing ofwhether the monitored vehicle is located on one of the blind spots by amultilayer perceptron to which said pieces of cue information fortesting or their processed values on the monitored vehicle are inputted.

As one example, said pieces of cue information for testing on themonitored vehicle include at least part of (i) the class information fortesting on the monitored vehicle, (ii) the location information fortesting on the monitored vehicle, (iii) size information for testing onthe monitored vehicle corresponding to an ROI size, (iv) aspect ratioinformation for testing on the monitored vehicle, and (v) distanceinformation for testing between a center of the monitored vehicle andone of outer sides of the blind spots.

As one example, the cue information extracting layer determines asmaller value among either of (i) a distance of one outer boundary ofthe blind spot BS-L from the center of the monitored vehicle and (ii) adistance of the other outer boundary of the blind spot BS-R from thecenter thereof as the distance information for testing between thecenter of the monitored vehicle and said one of the outer sides of theblind spots, and further outputs relative location information fortesting in addition to the distance information for testing in order todetermine whether the center of the monitored vehicle is located insideor outside of said one of the outer sides of the blind spots.

As one example, the monitoring vehicle for the testing device is notidentical to the monitoring vehicle for the learning device.

In accordance with still another aspect of the present invention, thereis provided a learning device of a CNN (Convolutional Neural Network)for monitoring one or more blind spots of a monitoring vehicle,including: a communication part for receiving a training imagecorresponding to at least one video image taken from the monitoringvehicle; and a processor for (I) instructing a detector on themonitoring vehicle to output class information and location informationon a monitored vehicle included in the training image, (II) instructinga cue information extracting layer to perform one or more operations byusing the class information and the location information on themonitored vehicle, to thereby output one or more pieces of cueinformation on the monitored vehicle, and instructing an FC layer formonitoring the blind spots to perform one or more neural networkoperations by using said pieces of cue information or their processedvalues on the monitored vehicle, to thereby output a result ofdetermining whether the monitored vehicle is located on one of the blindspots of the monitoring vehicle, and (III) instructing a first losslayer to generate one or more loss values for the blind spots byreferring to the result and its corresponding first GT (Ground Truth),to thereby learn one or more parameters of the FC layer for monitoringthe blind spots by backpropagating the loss values for the blind spots,and instructing a second loss layer to generate one or more loss valuesfor vehicle detection by referring to the class information and thelocation information on the monitored vehicle and their correspondingsecond GT, to thereby learn one or more parameters of the detector bybackpropagating the loss values for the vehicle detection.

As one example, the detector is a vehicle detector based on an R-CNN(Region-based Convolutional Neural Network) including: one or moreconvolutional layers which generate a feature map from the trainingimage, an RPN (Region Proposal Network) which generates an ROI (RegionOf Interest) of the monitored vehicle from the feature map, a poolinglayer which generates a feature vector by pooling an area, in thefeature map, corresponding to the ROI, at least one FC layer for thevehicle detection which performs at least one fully connected operationon the feature vector, to thereby generate one or more FC output values,a classification layer which outputs the class information on themonitored vehicle by referring to the FC output values, and a regressionlayer which outputs the location information on the monitored vehicle byreferring to the FC output values.

As one example, the loss values for the vehicle detection include one ormore class loss values and one or more location loss values of themonitored vehicle, wherein the processor learns one or more parametersof the FC layer for the vehicle detection and one or more parameters ofthe convolutional layers by backpropagating the class loss values andthe location loss values.

As one example, the processor instructs the CNN to receive aconcatenated value acquired by using said pieces of cue information onthe monitored vehicle generated by the cue information extracting layerand the feature vector generated by the pooling layer of the detector,as an input to the FC layer for monitoring the blind spots.

As one example, the FC layer for monitoring the blind spots includes aneural network capable of outputting a result value of whether themonitored vehicle is located on one of the blind spots by a multilayerperceptron to which said pieces of cue information on the monitoredvehicle are inputted.

As one example, said pieces of cue information on the monitored vehicleinclude at least part of (i) the class information on the monitoredvehicle, (ii) the location information on the monitored vehicle, (iii)size information on the monitored vehicle corresponding to an ROI(Region Of Interest) size, (iv) aspect ratio information on themonitored vehicle, and (v) distance information between a center of themonitored vehicle and one of outer sides of the blind spots.

As one example, the cue information extracting layer determines asmaller value among either of (i) a distance of one outer boundary ofthe blind spot BS-L from the center of the monitored vehicle and (ii) adistance of the other outer boundary of the blind spot BS-R from thecenter thereof as the distance information between the center of themonitored vehicle and said one of the outer sides of the blind spots,and further outputs relative location information in addition to thedistance information in order to determine whether the center of themonitored vehicle is located inside or outside of said one of the outersides of the blind spots.

In accordance with still yet another aspect of the present invention,there is provided a testing device of a CNN (Convolutional NeuralNetwork) for monitoring one or more blind spots of a monitoring vehicle,including: a communication part for, on condition that a learningdevice, (i) has instructed a detector on the monitoring vehicle tooutput class information for training and location information fortraining on a monitored vehicle included in a training imagecorresponding to at least one video image taken from the monitoringvehicle, (ii) has instructed a cue information extracting layer toperform one or more operations by using the class information fortraining and the location information for training on the monitoredvehicle, to thereby output one or more pieces of cue information fortraining on the monitored vehicle, (iii) has instructed an FC layer formonitoring the blind spots to perform one or more neural networkoperations by using said pieces of cue information for training or theirprocessed values, to thereby output a result for training of determiningwhether the monitored vehicle is located on one of the blind spots, and(iv) has instructed a first loss layer to generate one or more lossvalues for the blind spots by referring to the result for training andits corresponding first GT (Ground Truth), to thereby learn one or moreparameters of the FC layer for monitoring the blind spots bybackpropagating the loss values for the blind spots, and has instructeda second loss layer to generate one or more loss values for vehicledetection by referring to the class information for training and thelocation information for training and their corresponding second GT, tothereby learn one or more parameters of the detector by backpropagatingthe loss values for the vehicle detection, acquiring class informationfor testing and location information for testing on the monitoredvehicle form the detector which detects the monitored vehicle in a testimage taken from the monitoring vehicle; and a processor for (I)instructing the cue information extracting layer to perform theoperations by using the class information for testing and the locationinformation for testing on the monitored vehicle, to thereby output oneor more pieces of cue information for testing on the monitored vehicle,and (II) instructing the FC layer for monitoring the blind spots toperform the neural network operations by using said pieces of cueinformation for testing or their processed values on the monitoredvehicle, to thereby output a result for testing of determining whetherthe monitored vehicle is located on one of the blind spots of themonitoring vehicle.

As one example, the detector is a vehicle detector based on an R-CNN(Region-based Convolutional Neural Network) including: one or moreconvolutional layers which generate a feature map for testing from thetest image, an RPN (Region Proposal Network) which generates an ROI(Region Of Interest) for testing of the monitored vehicle from thefeature map for testing, a pooling layer which generates a featurevector for testing by pooling an area, in the feature map for testing,corresponding to the ROI for testing, at least one FC layer for thevehicle detection which performs at least one fully connected operationon the feature vector for testing, to thereby generate one or more FCoutput values, a classification layer which outputs the classinformation for testing on the monitored vehicle by referring to the FCoutput values, and a regression layer which outputs the locationinformation for testing on the monitored vehicle by referring to the FCoutput values.

As one example, the processor instructs the CNN to receive aconcatenated value for testing acquired by using said pieces of cueinformation for testing on the monitored vehicle generated by the cueinformation extracting layer and the feature vector for testinggenerated by the pooling layer of the detector, as an input to the FClayer for monitoring the blind spots.

As one example, the FC layer for monitoring the blind spots includes aneural network capable of outputting a result value for testing ofwhether the monitored vehicle is located on one of the blind spots by amultilayer perceptron to which said pieces of cue information fortesting or their processed values on the monitored vehicle are inputted.

As one example, said pieces of cue information for testing on themonitored vehicle include at least part of (i) the class information fortesting on the monitored vehicle, (ii) the location information fortesting on the monitored vehicle, (iii) size information for testing onthe monitored vehicle corresponding to an ROI size, (iv) aspect ratioinformation for testing on the monitored vehicle, and (v) distanceinformation for testing between a center of the monitored vehicle andone of outer sides of the blind spots.

As one example, the cue information extracting layer determines asmaller value among either of (i) a distance of one outer boundary ofthe blind spot BS-L from the center of the monitored vehicle and (ii) adistance of the other outer boundary of the blind spot BS-R from thecenter thereof as the distance information for testing between thecenter of the monitored vehicle and said one of the outer sides of theblind spots, and further outputs relative location information fortesting in addition to the distance information for testing in order todetermine whether the center of the monitored vehicle is located insideor outside of said one of the outer sides of the blind spots.

As one example, the monitoring vehicle for the testing device is notidentical to the monitoring vehicle for the learning device.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings attached below to explain example embodiments of thepresent invention are only part of example embodiments of the presentinvention and other drawings may be obtained based on the drawingswithout inventive work for those skilled in the art:

FIG. 1 is a drawing schematically illustrating a learning device formonitoring blind spots of a monitoring vehicle in accordance with oneexample embodiment of the present invention.

FIG. 2 is a drawing schematically illustrating a learning method formonitoring the blind spots in accordance with one example embodiment ofthe present invention.

FIG. 3 is a drawing schematically illustrating location information on amonitored vehicle among one or more pieces of cue information on themonitored vehicle in the learning method for monitoring the blind spotsin accordance with one example embodiment of the present invention.

FIG. 4 is a drawing schematically illustrating distance informationbetween a center of the monitored vehicle and one of outer sides of theblind spots among said pieces of cue information on the monitoredvehicle in the learning method for monitoring the blind spots inaccordance with one example embodiment of the present invention.

FIG. 5 is a drawing schematically illustrating an FC layer formonitoring the blind spots in the learning method for monitoring theblind spots in accordance with one example embodiment of the presentinvention.

FIG. 6 is a drawing schematically illustrating a testing device formonitoring the blind spots in accordance with one example embodiment ofthe present invention.

FIG. 7 is a drawing schematically illustrating a testing method formonitoring the blind spots in accordance with one example embodiment ofthe present invention.

FIG. 8 is a diagram schematically illustrating a detector used in thetesting device for monitoring the blind spots in accordance with oneexample embodiment of the present invention.

FIG. 9 is a drawing schematically illustrating a learning device formonitoring blind spots of a monitoring vehicle in accordance withanother example embodiment of the present invention.

FIG. 10 is a diagram schematically illustrating a learning method formonitoring the blind spots in accordance with another example embodimentof the present invention.

FIG. 11 is a drawing schematically illustrating a testing device formonitoring the blind spots in accordance with another example embodimentof the present invention.

FIG. 12 is a diagram schematically illustrating a testing method formonitoring the blind spots in accordance with another example embodimentof the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following detailed description, reference is made to theaccompanying drawings that show, by way of illustration, specificembodiments in which the invention may be practiced. These embodimentsare described in sufficient detail to enable those skilled in the art topractice the invention.

Besides, in the detailed description and claims of the presentinvention, a term “include” and its variations are not intended toexclude other technical features, additions, components or steps. Otherobjects, benefits, and features of the present invention will berevealed to one skilled in the art, partially from the specification andpartially from the implementation of the present invention. Thefollowing examples and drawings will be provided as examples but theyare not intended to limit the present invention.

Moreover, the present invention covers all possible combinations ofexample embodiments indicated in this specification. It is to beunderstood that the various embodiments of the present invention,although different, are not necessarily mutually exclusive. For example,a particular feature, structure, or characteristic described herein inconnection with one embodiment may be implemented within otherembodiments without departing from the spirit and scope of the presentinvention. In addition, it is to be understood that the position orarrangement of individual elements within each disclosed embodiment maybe modified without departing from the spirit and scope of the presentinvention. The following detailed description is, therefore, not to betaken in a limiting sense, and the scope of the present invention isdefined only by the appended claims, appropriately interpreted, alongwith the full range of equivalents to which the claims are entitled. Inthe drawings, like numerals refer to the same or similar functionalitythroughout the several views.

To allow those skilled in the art to the present invention to be carriedout easily, the example embodiments of the present invention byreferring to attached drawings will be explained in detail as shownbelow.

FIG. 1 is a drawing schematically illustrating a learning device 100 formonitoring blind spots of a monitoring vehicle in accordance with oneexample embodiment of the present invention and, by referring to FIG. 1,the learning device 100 may include a communication part 110 and aprocessor 120.

First, the communication part 110 may receive training datacorresponding to at least one output signal from a detector on themonitoring vehicle. That is, the communication part 110 may receive theoutput signal from the detector in following detailed description as aninput signal, i.e., the training data.

Herein, the training data, as information on monitored vehicles in animage corresponding to an input video from the detector, may includeclass information on objects such as the monitored vehicles and locationinformation on areas where the monitored vehicles are located in theimage. Additionally, the training data may be stored in a database 130where GT, i.e., ground truth, of the class information and the locationinformation on each of the monitored vehicles corresponding to thetraining data may be stored.

Next, the processor 120 may perform a first process of instructing a cueinformation extracting layer to perform one or more operations by usingthe class information and the location information on a monitoredvehicle included in the training data, to thereby output one or morepieces of cue information on the monitored vehicle, a second process ofinstructing an FC layer for monitoring the blind spots to perform one ormore neural network operations by using said pieces of cue informationon the monitored vehicle, to thereby output a result of determiningwhether the monitored vehicle is located on one of the blind spots ofthe monitoring vehicle, and a third process of instructing a loss layerto generate one or more loss values by referring to the result and itscorresponding GT, to thereby learn one or more parameters of the FClayer for monitoring the blind spots by backpropagating the loss values.

Herein, the learning device 100 in accordance with one exampleembodiment of the present invention may be a computing device and mayinclude any device having a processor capable of computation inaccordance with the present invention. Also, FIG. 1 represents onelearning device 100 but the scope of the present invention is notlimited thereto. That is, the learning device 100 may be configured asseveral devices to perform its functions.

A learning method for monitoring the blind spot of the monitoringvehicle by using the learning device configured as aforementioned inaccordance with one example embodiment of the present invention may beexplained as below, by referring to FIG. 2.

First of all, if the training data corresponding to the output signalfrom the detector on the monitoring vehicle is inputted, the learningdevice 100 may instruct the cue information extracting layer 121 toperform the operations by using the class information and the locationinformation on the monitored vehicle included in the training data, tothereby output said pieces of cue information on the monitored vehicle.

Herein, the detector capable of detecting the monitored vehicle in animage acquired by a vision sensor may be configured with a hypothesisgeneration stage of detecting an ROI (Region Of Interest), i.e., an areawhere the monitored vehicle is presumed to be located in the image, anda hypothesis verification stage of determining whether the detected ROIactually includes the monitored vehicle. Also, the hypothesis generationstage may be implemented with a motion-based scheme using optical flows,an appearance-based scheme using shadow underneath vehicles, corners,vertical and horizontal edges, symmetry, color, vehicle's lights, stereocameras, multiple features, or the like, and the hypothesis verificationstage may be implemented with a correlation-based scheme using templatematching, a learning-based scheme using features and classifiers, or thelike. Especially, the learning-based scheme may adopt a deep learningbased algorithm such as a CNN, or a shallow learning based algorithmsuch as a decision tree, an SVM (Support Vector Machine), an AdaBoost,k-nearest neighbors, etc.

Moreover, said pieces of cue information on the monitored vehicle mayinclude at least part of (i) the class information on the monitoredvehicle, (ii) the location information on the monitored vehicle, (iii)size information on the monitored vehicle corresponding to an ROI size,(iv) aspect ratio information on the monitored vehicle, and (v) distanceinformation between a center of the monitored vehicle and one of outersides of the blind spots.

Herein, the class information on the monitored vehicle may include classinformation for classifying the monitored vehicle as a car, amotorcycle, etc.

Also, by referring to FIG. 3, the location information on the monitoredvehicle may include information on a location corresponding to an areawhere the monitored vehicle is located in the image. For example,coordinates of the top-left and the bottom-right corners of a boundingbox in the image may be included, as information on a location of thebounding box corresponding to the monitored vehicle. Coordinates of acenter of the bounding box, which may be calculated by using thecoordinates of the top-left and the bottom-right corners of the boundingbox, may further be included.

Additionally, by referring to FIG. 4, a smaller value among either of(i) a distance of one outer boundary of the blind spot BS-L from thecenter of the monitored vehicle and (ii) a distance of the other outerboundary of the blind spot BS-R from the center thereof may bedetermined as the distance information L. Herein, the distanceinformation L may further include relative location information, e.g.,information on plus sign or negative sign, to determine whether thecenter of the monitored vehicle is located inside or outside of said oneof outer sides of the blind spots.

Next, the learning device 100 may instruct the FC layer 122 formonitoring the blind spots to perform the neural network operations byusing said pieces of the cue information on the monitored vehicle, tothereby output the result of determining whether the monitored vehicleis located on said one of the blind spots of the monitoring vehicle.

By referring to FIG. 5, the FC layer 122 for monitoring the blind spotsmay include a neural network capable of outputting a result value ofwhether the monitored vehicle is located on said one of the blind spotsby a multilayer perceptron to which said pieces of cue information onthe monitored vehicle are inputted. In FIG. 5, the number of inputs isrepresented as four, i.e., input1, input2, input3, and input4, forconvenience of explanation, but the number of the inputs may bedetermined as corresponding to said pieces of cue information generatedby the cue information extracting layer 121. Also, although a hiddenlayer is represented by one layer in FIG. 5, the scope of the presentinvention is not limited thereto, and the hidden layer may includemultiple layers.

Next, the learning device 100 may instruct the loss layer 123 togenerate the loss values by referring to the result from the FC layer122 for monitoring the blind spots and its corresponding GT, to therebylearn the parameters of the FC layer 122 by backpropagating the lossvalues.

The cue information extracting layer 121, the FC layer 122, and the losslayer 123 may be included in one computing device or in differentcomputing devices, and may also be implemented as an algorithmperforming the operations previously mentioned, in any computingdevices.

FIG. 6 is a drawing schematically illustrating a testing device 200 formonitoring blind spots of a monitoring vehicle, which may or may not bethe same as the monitoring vehicle mentioned previously, in accordancewith one example embodiment of the present invention, and the testingdevice 200 may include a communication part 210 and a processor 220 byreferring to FIG. 6.

First of all, the communication part 210 may receive class informationfor testing and location information for testing on a monitored vehiclefrom a detector 20 which detects the monitored vehicle in a test imagetaken from the monitoring vehicle.

Next, the processor 220 may perform a first process of instructing thecue information extracting layer to perform the operations by using theclass information for testing and the location information for testingon the monitored vehicle, to thereby output one or more pieces of cueinformation for testing on the monitored vehicle, and a second processof instructing the FC layer for monitoring the blind spots to performthe neural network operations by using said pieces of cue informationfor testing or their processed values on the monitored vehicle, tothereby output a result for testing of whether the monitored vehicle islocated on one of the blind spots of the monitoring vehicle.

Hereinafter, “for training” and “for testing” may be used to distinguishthe testing method from the aforementioned learning method.

Herein, the FC layer for monitoring the blind spots has learned one ormore parameters of the learning device using the learning methodexplained by referring to FIGS. 1 to 5, and the learning method may bebriefly explained as follows. The learning device, (i) if the trainingdata corresponding to at least one output signal from the detector 20 onthe monitoring vehicle has been inputted, has instructed the cueinformation extracting layer to perform the operations by using classinformation for training and location information for training on themonitored vehicle included in the training data, to thereby output oneor more pieces of cue information for training on the monitored vehicle,(ii) has instructed the FC layer for monitoring the blind spots toperform the neural network operations by using said pieces of cueinformation for training on the monitored vehicle, to thereby output aresult for training of determining whether the monitored vehicle islocated on said one of the blind spots of the monitoring vehicle, and(iii) has instructed the loss layer to generate one or more loss valuesby referring to the result for training and its corresponding GT, tothereby learn the parameters of the FC layer for monitoring the blindspots by backpropagating the loss values.

Additionally, the testing device 200 in accordance with one exampleembodiment of the present invention may be a computing device and mayinclude any device having a processor capable of computation inaccordance with the present invention. Also, FIG. 6 represents onetesting device 200 but the scope of the present invention is not limitedthereto. That is, the testing device 200 may be configured as severaldevices to perform its functions.

A testing method for monitoring the blind spots using the testing devicefor monitoring the blind spots configured as aforementioned inaccordance with one example embodiment of the present invention may beexplained as below, by referring to FIG. 7.

First of all, before performing the testing method, on condition thatthe learning device (i) has instructed the cue information extractinglayer 221 to perform the operations by using the class information fortraining and the location information for training on the monitoredvehicle included in the training data corresponding to the output signalfrom the detector 20, to thereby output said pieces of cue informationon the monitored vehicle, (ii) has instructed the FC layer formonitoring the blind spots to perform the neural network operations byusing said pieces of cue information, to thereby output the result fortraining whether the monitored vehicle is located on the blind spots ofthe monitoring vehicle, and (iii) has instructed the loss layer togenerate the loss values by referring to the result for training and itscorresponding GT, to thereby learn the parameters of the FC layer formonitoring the blind spots by backpropagating the loss values, thedetector 20 may detect the monitored vehicle in the test image takenfrom the monitoring vehicle and may output the class information fortesting and the location information for testing on the monitoredvehicle.

Herein, the detector 20 capable of detecting the monitored vehicle in animage acquired by a vision sensor may be configured with a hypothesisgeneration stage of detecting an ROI, i.e., an area where the monitoredvehicle is presumed to be located in the image, and a hypothesisverification stage of determining whether the detected ROI actuallyincludes the monitored vehicle. Also, the hypothesis generation stagemay be implemented with a motion-based scheme using optical flows, anappearance-based scheme using shadow underneath vehicles, corners,vertical and horizontal edges, symmetry, color, vehicle's lights, stereocameras, multiple features, or the like, and the hypothesis verificationstage may be implemented with a correlation-based scheme using templatematching, a learning-based scheme using features and classifiers, or thelike. Especially, the learning-based scheme may adopt a deep learningbased algorithm, such as a CNN, or a shallow learning based algorithm,such as a decision tree, an SVM (Support Vector Machine), an AdaBoost,k-nearest neighbors, etc.

As one example, by referring to FIG. 8, the detector 20 may be a vehicledetector based on an R-CNN (Region-based Convolutional Neural Network)including a convolutional layer 21 which generates a feature map fortesting from the test image, an RPN (Region Proposal Network) 22 whichgenerates an ROI for testing of the monitored vehicle from the featuremap for testing, a pooling layer 23 which generates a feature vector fortesting by pooling an area, in the feature map for testing,corresponding to the ROI for testing, at least one FC layer 24 forvehicle detection which performs at least one fully connected operationon the feature vector for testing, to thereby generate one or more FCoutput values, a classification layer 25 which outputs the classinformation for testing on the monitored vehicle by referring to the FCoutput values, and a regression layer 26 which outputs the locationinformation for testing on the monitored vehicle by referring to the FCoutput values. Herein, each of the convolutional layer 21 and the FClayer 24 is presented as one layer, but the scope of the presentinvention is not limited thereto and each of them may be comprised ofmultiple layers. Also, the feature map for testing may be outputted inform of one or more feature maps corresponding to a depth of a channel,from the convolutional layer 21.

Then, the testing device 200 may instruct the cue information extractinglayer 221 to perform the operations by using the class information fortesting and the location information for testing on the monitoredvehicle, to thereby output said pieces of cue information for testing onthe monitored vehicle.

Herein, said pieces of cue information for testing on the monitoredvehicle may include at least part of (i) the class information fortesting on the monitored vehicle, (ii) the location information fortesting on the monitored vehicle, (iii) size information for testing onthe monitored vehicle corresponding to an ROI size for testing, (iv)aspect ratio information for testing on the monitored vehicle, and (v)distance information for testing between a center of the monitoredvehicle and one of outer sides of the blind spots.

Also, the class information for testing on the monitored vehicle mayinclude class information for classifying the monitored vehicle as acar, a motorcycle, etc.

In addition, as explained by referring to FIG. 3, the locationinformation for testing on the monitored vehicle may include informationon a location corresponding to an area where the monitored vehicle islocated in the test image. For example, coordinates of the top-left andthe bottom-right corners of a bounding box in the test image may beincluded, as information on a location of the bounding box correspondingto the monitored vehicle. Coordinates of a center of the bounding box,which may be calculated by using the coordinates of the top-left and thebottom-right corners of the bounding box, may be further included.

Additionally, as explained by referring to FIG. 4, a smaller value amongeither of (i) a distance of one outer boundary of the blind spot BS-Lfrom the center of the monitored vehicle and (ii) a distance of theother outer boundary of the blind spot BS-R from the center thereof maybe determined as the distance information L for testing. Herein, thedistance information L for testing may include relative locationinformation for testing, e.g., information on plus sign or negativesign, to determine whether the center of the monitored vehicle islocated inside or outside of said one of outer sides of the blind spots.

Next, the testing device 200 may instruct the FC layer 222 formonitoring the blind spots to perform the neural network operations byusing said pieces of the cue information for testing or their processedvalues on the monitored vehicle, to thereby output a result for testingof determining whether the monitored vehicle is located on said one ofthe blind spots of the monitoring vehicle.

Herein, the processed values of said pieces of cue information fortesting may represent a concatenated value acquired by using said piecesof cue information for testing on the monitored vehicle generated by thecue information extracting layer 221 and the feature vector for testinggenerated by the pooling layer 23 of the detector 20. A featureconcatenation layer capable of concatenating said pieces of cueinformation for testing and the feature vector for testing may belocated before the FC layer 222 for monitoring the blind spots.

Also, as explained by referring to FIG. 5, the FC layer 222 may includethe neural network capable of outputting a result value of whether themonitored vehicle is located on said one of the blind spots by themultilayer perceptron to which said pieces of cue information fortesting or their processed values on the monitored vehicle are inputted.

The cue information extracting layer 221 and the FC layer 222 formonitoring the blind spots may be included in one computing device or indifferent computing devices, and may also be implemented as an algorithmperforming the operations previously mentioned, in any computingdevices.

FIG. 9 is a drawing schematically illustrating a learning device 300 formonitoring blind spots of a monitoring vehicle in accordance withanother example embodiment of the present invention, and the learningdevice 300 may include a communication part 310 and a processor 320 byreferring to FIG. 9.

First of all, the communication part 310 may receive a training imagecorresponding to at least one video image taken from the monitoringvehicle.

Herein, the training image may be stored in a database 330 where atleast one GT of class information and location information on amonitored vehicle corresponding to the training image may be stored.

Next, the processor 320 may perform a first process of instructing adetector 30 on the monitoring vehicle to output the class informationand the location information on the monitored vehicle, a second processof instructing a cue information extracting layer to perform one or moreoperations by using the class information and the location informationon the monitored vehicle, to thereby output one or more pieces of cueinformation on the monitored vehicle, and instructing an FC layer formonitoring the blind spots to perform one or more neural networkoperations by using said pieces of cue information or their processedvalues on the monitored vehicle, to thereby output a result ofdetermining whether the monitored vehicle is located on one of the blindspots, and a third process of instructing a first loss layer to generateone or more loss values for the blind spots by referring to the resultand its corresponding first GT, to thereby learn one or more parametersof the FC layer for monitoring the blind spots by backpropagating theloss values for the blind spots, and instructing a second loss layer togenerate one or more loss values for vehicle detection by referring tothe class information and the location information on the monitoredvehicle and their corresponding second GT, to thereby learn one or moreparameters of the detector by backpropagating the loss values for thevehicle detection.

Herein, the learning device 300 in accordance with another exampleembodiment of the present invention may be a computing device and mayinclude any device having a processor capable of computation inaccordance with the present invention. Also, FIG. 9 represents onelearning device 300 but the scope of the present invention is notlimited thereto. That is, the learning device 300 may be configured asseveral devices to perform its functions.

A learning method for monitoring the blind spot of the monitoringvehicle by using the learning device configured as aforementioned inaccordance with another example embodiment of the present invention maybe explained as below, by referring to FIG. 10.

First, if the training image corresponding to the video image taken fromthe monitoring vehicle is inputted, the detector 30 may output the classinformation and the location information on the monitored vehicleincluded in the training image.

Herein, the detector 30 capable of detecting the monitored vehicle in animage acquired by a vision sensor may be configured with a hypothesisgeneration stage of detecting an ROI, i.e., an area where the monitoredvehicle is presumed to be located in the image, and a hypothesisverification stage of determining whether the detected ROI actuallyincludes the monitored vehicle. Also, the hypothesis generation stagemay be implemented with a motion-based scheme using optical flows, anappearance-based scheme using shadow underneath vehicles, corners,vertical and horizontal edges, symmetry, color, vehicle's lights, stereocameras, multiple features, or the like, and the hypothesis verificationstage may be implemented with a correlation-based scheme using templatematching, a learning-based scheme using features and classifiers, or thelike. Especially, the learning-based scheme may adopt a deep learningbased algorithm, such as a CNN, or a shallow learning based algorithm,such as a decision tree, an SVM (Support Vector Machine), an AdaBoost,k-nearest neighbors, etc.

As one example, similarly to the detector explained by referring to FIG.8, the detector 30 may be a vehicle detector based on an R-CNN,including a convolutional layer 31 which generates a feature map fromthe training image, an RPN 32 which generates the ROI of the monitoredvehicle from the feature map, a pooling layer 33 which generates afeature vector by pooling an area, in the feature map, corresponding tothe ROI, and at least one FC layer 34 for the vehicle detection whichperforms at least one fully connected operation on the feature vector tothereby generate one or more FC output values, a classification layer 35which outputs the class information on the monitored vehicle byreferring to the FC output values, and a regression layer 36 whichoutputs the location information on the monitored vehicle by referringto the FC output values. Herein, the convolutional layer 31 may becomprised of at least one layer and the FC layer 34 may also becomprised of at least one layer.

Next, the learning device 300 may instruct the cue informationextracting layer 321 to perform the operations by using the classinformation and the location information on the monitored vehicle, tothereby output said pieces of cue information on the monitored vehicle.

Herein, said pieces of cue information on the monitored vehicle mayinclude at least part of (i) the class information on the monitoredvehicle, (ii) the location information on the monitored vehicle, (iii)size information on the monitored vehicle corresponding to an ROI size,(iv) aspect ratio information on the monitored vehicle, and (v) distanceinformation between a center of the monitored vehicle and one of outersides of the blind spots.

Also, the class information on the monitored vehicle may include classinformation for classifying the monitored vehicle as a car, amotorcycle, etc.

Additionally, similarly as explained by referring to FIG. 3, thelocation information on the monitored vehicle may include information ona location corresponding to an area where the monitored vehicle islocated in the image. For example, coordinates of the top-left and thebottom-right corners of a bounding box in the image may be included, asinformation on a location of the bounding box corresponding to themonitored vehicle. Coordinates of a center of the bounding box, whichmay be calculated by using the coordinates of the top-left and thebottom-right corners of the bounding box, may be further included.

Additionally, similarly as explained by referring to FIG. 4, a smallervalue among either of (i) a distance of one outer boundary of the blindspot BS-L from the center of the monitored vehicle and (ii) a distanceof the other outer boundary of the blind spot BS-R from the centerthereof may be determined as the distance information L. Herein, thedistance information L may include relative location information, e.g.,information on plus sign or negative sign, to determine whether thecenter of the monitored vehicle is located inside or outside of said oneof outer sides of the blind spots.

Next, the learning device 300 may instruct the FC layer 322 formonitoring the blind spots to perform the neural network operations byusing said pieces of the cue information or their processed values onthe monitored vehicle, to thereby output the result of determiningwhether the monitored vehicle is located on said one of the blind spotsof the monitoring vehicle.

Herein, the processed values of said pieces of cue information mayrepresent a concatenated value acquired by using said pieces of cueinformation on the monitored vehicle generated by the cue informationextracting layer 321 and the feature vector generated by the poolinglayer 33 of the detector 30. A feature concatenation layer 340, whichgenerates the concatenated value including said pieces of cueinformation and the feature vector, may be located before the FC layer322 for monitoring the blind spots.

Also, similarly as explained by referring to FIG. 5, the FC layer 322may include a neural network capable of outputting a result value ofwhether the monitored vehicle is located on said one of the blind spotsby a multilayer perceptron to which said pieces of cue information ortheir processed values on the monitored vehicle are inputted.

Next, the learning device 300 may instruct the first loss layer 323 togenerate the loss values for the blind spots by referring to (i) theresult of determining whether the monitored vehicle is located on saidone of the blind spots of the monitoring vehicle and (ii) itscorresponding first GT, to thereby learn the parameters of the FC layer322 for monitoring the blind spots by backpropagating the loss valuesfor the blind spots, and instructing the second loss layer 324 togenerate the loss values for the vehicle detection by referring to theclass information and the location information on the monitored vehicleand their corresponding second GT, to thereby learn the parameters ofthe detector 30 by backpropagating the loss values for the vehicledetection.

Herein, the loss values for the vehicle detection may include one ormore class loss values and one or more location loss values of themonitored vehicle, and the learning device 300 may learn one or moreparameters of the FC layer 34 for the vehicle detection and one or moreparameters of the convolutional layer 31 of the detector 30 bybackpropagating the class loss values and the location loss values.Also, although not illustrated, the learning device 300 may acquire oneor more ROI loss values by referring to the ROI, which is generated fromthe RPN 32 of the detector 30, and ROI GT corresponding to the trainingimage, to thereby learn one or more parameters of the RPN 32 bybackpropagating the ROI loss values.

The cue information extracting layer 221 and the FC layer 222 formonitoring the blind spots may be included in one computing device or indifferent computing devices, and may also be implemented as an algorithmperforming the operations previously mentioned, in any computingdevices. Additionally, the first loss layer 323 and the second losslayer 324 may be included in one computing device or in differentcomputing devices respectively and may be implemented as an algorithmperforming the operations previously mentioned, in any computingdevices.

FIG. 11 is a drawing schematically illustrating a testing device 400 formonitoring blind spots of a monitoring vehicle, which may or may not bethe same as the monitoring vehicle mentioned previously, in accordancewith another example embodiment of the present invention, and thetesting device 400 may include a communication part 410 and a processor420 by referring to FIG. 11.

First of all, the communication part 410 may receive class informationfor testing and location information for testing on a monitored vehiclefrom a detector 40 which detects the monitored vehicle in a test imagetaken from the monitoring vehicle.

Herein, the detector 40 capable of detecting the monitored vehicle in animage acquired by a vision sensor may be configured with a hypothesisgeneration stage of detecting an ROI, i.e., an area where the monitoredvehicle is presumed to be located in the image and a hypothesisverification stage of determining whether the detected ROI actuallyincludes the monitored vehicle. Also, the hypothesis generation stagemay be implemented with a motion-based scheme using optical flows, anappearance-based scheme using shadow underneath vehicles, corners,vertical and horizontal edges, symmetry, color, vehicle's lights, stereocameras, multiple features, or the like, and the hypothesis verificationstage may be implemented with a correlation-based scheme using templatematching, a learning-based scheme using features and classifiers, or thelike. Especially, the learning-based scheme may adopt a deep learningbased algorithm, such as a CNN, or a shallow learning based algorithm,such as a decision tree, an SVM (Support Vector Machine), an AdaBoost,k-nearest neighbors, etc.

As one example, similarly to the detector explained by referring to FIG.8, the detector 40 may be a vehicle detector based on the R-CNN,including the convolutional layer which generates a feature map fortesting from the test image, the RPN which generates an ROI for testingof the monitored vehicle from the feature map for testing, the poolinglayer which generates a feature vector for testing by pooling an area,in the feature map for testing, corresponding to the ROI for testing,the FC layer for the vehicle detection which performs the fullyconnected operations by using the feature vector for testing, to therebygenerate one or more FC output values, the classification layer whichoutputs the class information for testing on the monitored vehicle byreferring to the FC output values, and the regression layer whichoutputs the location information for testing on the monitored vehicle byreferring to the FC output values. Herein, each of the convolutionallayer and the FC layer for the vehicle detection is presented as onelayer, but the scope of the present invention is not limited thereto andeach of them may be comprised of multiple layers. Also, the feature mapfor testing may be outputted in form of one or more feature mapscorresponding to a depth of a channel, from the convolutional layer 41.

Next, the processor 420 may perform a first process of instructing thecue information extracting layer to perform the operations by using theclass information for testing and the location information for testingon the monitored vehicle, to thereby output one or more pieces of cueinformation for testing on the monitored vehicle, and a second processof instructing the FC layer for monitoring the blind spots to performthe neural network operations by using said pieces of cue informationfor testing or their processed values on the monitored vehicle, tothereby output a result for testing of whether the monitored vehicle islocated on one of the blind spots of the monitoring vehicle.

Hereinafter, “for training” and “for testing” may be used to distinguishthe testing method from the aforementioned learning method.

Herein, the detector 40 and the FC layer for monitoring the blind spotshave learned the parameters of the learning device using the learningmethod explained by referring to FIGS. 9 and 10, and the learning methodmay be briefly explained as follows. If the detector 40 has outputtedclass information for training and location information for training onthe monitored vehicle in a training image corresponding to a video imagetaken from the monitoring vehicle, the learning device (i) hasinstructed the cue information extracting layer to perform theoperations by using the class information for training and the locationinformation for training on the monitored vehicle, to thereby output oneor more pieces of cue information for training on the monitored vehicle,(ii) has instructed the FC layer for monitoring the blind spots toperform the neural network operations by using said pieces of cueinformation for training or their processed values, to thereby output aresult for training of whether the monitored vehicle is located on saidone of the blind spots, and (iii) has instructed the first loss layer togenerate one or more loss values for the blind spots by referring to theresult for training and its corresponding first GT, to thereby learn theparameters of the FC layer for monitoring the blind spots bybackpropagating the loss values for the blind spots, and has instructedthe second loss layer to generate one or more loss values for thevehicle detection by referring to the class information for training andthe location information for training and their corresponding second GT,to thereby learn the parameters of the detector 40 by backpropagatingthe loss values for the vehicle detection.

Additionally, the testing device 400 in accordance with another exampleembodiment of the present invention may be a computing device and mayinclude any device having a processor capable of computation inaccordance with the present invention. Also, FIG. 11 represents onetesting device 400 but the scope of the present invention is not limitedthereto. That is, the testing device 400 may be configured as severaldevices to perform its functions.

A testing method for monitoring the blind spots using the testing devicefor monitoring the blind spots configured as aforementioned inaccordance with another example embodiment of the present invention maybe explained as below, by referring to FIG. 12.

First of all, on condition that if the detector 40 has outputted theclass information for training and the location information for trainingon the monitored vehicle in the training image corresponding to thevideo image taken from the monitoring vehicle, the learning device (i)has instructed the cue information extracting layer 421 to perform theoperations by using the class information for training and locationinformation for training on the monitored vehicle, to thereby outputsaid pieces of cue information for training on the monitored vehicle,(ii) has instructed the FC layer 422 for monitoring the blind spots toperform the neural network operations by using said pieces of cueinformation for training or their processed values, to thereby outputthe result for training of whether the monitored vehicle is located onsaid one of the blind spots, and (iii) has instructed the first losslayer to generate the loss values for the blind spots by referring tothe result for training and its corresponding first GT, to thereby learnthe parameters of the FC layer 422 for monitoring the blind spots bybackpropagating the loss values for the blind spots, and has instructedthe second loss layer to generate the loss values for the vehicledetection by referring to the class information for training and thelocation information for training and their corresponding second GT, tothereby learn the parameters of the detector 40 by backpropagating theloss values for the vehicle detection, the detector 40 may detect themonitored vehicle in a test image taken from the monitoring vehicle, andoutput the class information for testing and location information fortesting on the monitored vehicle.

Herein, the detector 40 capable of detecting the monitored vehicle in animage acquired by the vision sensor may be configured with thehypothesis generation stage of detecting the ROI, i.e., the area wherethe monitored vehicle is presumed to be located in the image, and thehypothesis verification stage of determining whether the detected ROIactually includes the monitored vehicle. Also, the hypothesis generationstage may be implemented with a motion-based scheme using optical flows,an appearance-based scheme using shadow underneath vehicle, corners,vertical and horizontal edges, symmetry, color, vehicle's lights, stereocameras, multiple features, or the like, and the hypothesis verificationstage may be implemented with a correlation-based scheme using templatematching, a learning-based scheme using features and classifiers, or thelike. Especially, the learning-based scheme may adopt a deep learningbased algorithm, such as a CNN, or a shallow learning based algorithm,such as a decision tree, an SVM (Support Vector Machine), an AdaBoost,k-nearest neighbors, etc.

As one example, the detector 40 may be a vehicle detector based on theR-CNN, including the convolutional layer 41 which generates a featuremap for testing from the test image, the RPN 42 which generates the ROIfor testing of the monitored vehicle from the feature map for testing,the pooling layer 43 which generates a feature vector for testing bypooling an area, in the feature map for testing, corresponding to theROI for testing, the FC layer 44 for the vehicle detection whichperforms the fully connected operation on the feature vector fortesting, to thereby generate one or more FC output values, theclassification layer 45 which outputs the class information for testingon the monitored vehicle by referring to the FC output values, and theregression layer 46 which outputs the location information for testingon the monitored vehicle by referring to the FC output values. Herein,each of the convolutional layer 41 and the FC layer 44 for the vehicledetection is presented as one layer, but the scope of the presentinvention is not limited thereto and each of them may be comprised ofmultiple layers.

Then, the testing device 400 may instruct the cue information extractinglayer 421 to perform the operations by using the class information fortesting and the location information for testing on the monitoredvehicle, to thereby output said pieces of cue information for testing onthe monitored vehicle.

Herein, said pieces of cue information for testing on the monitoredvehicle may include at least part of (i) the class information fortesting on the monitored vehicle, (ii) the location information fortesting on the monitored vehicle, (iii) size information for testing onthe monitored vehicle corresponding to an ROI size for testing, (iv)aspect ratio information for testing on the monitored vehicle, and (v)distance information for testing between a center of the monitoredvehicle and one of outer sides of the blind spots.

Also, the class information for testing on the monitored vehicle mayinclude class information for classifying the monitored vehicle as acar, a motorcycle, etc.

Additionally, similarly as explained by referring to FIG. 3, thelocation information for testing on the monitored vehicle may includeinformation on a location corresponding to an area where the monitoredvehicle is located in the test image. For example, coordinates of thetop-left and the bottom-right corners of a bounding box in the testimage may be included, as information on a location of the bounding boxcorresponding to the monitored vehicle. Coordinates of a center of thebounding box, which may be calculated by using the coordinates of thetop-left and the bottom-right corners of the bounding box, may befurther included.

Additionally, similarly as explained by referring to FIG. 4, a smallervalue among either of (i) a distance of one outer boundary of the blindspot BS-L from the center of the monitored vehicle and (ii) a distanceof the other outer boundary of the blind spot BS-R from the centerthereof may be determined as the distance information L for testing.Herein, the distance information L for testing may include relativelocation information for testing, e.g., information on plus sign ornegative sign, to determine whether the center of the monitored vehicleis located inside or outside of said one of outer sides of the blindspots.

Next, the testing device 400 may instruct the FC layer 422 formonitoring the blind spots to perform the neural network operations byusing said pieces of the cue information for testing or their processedvalues on the monitored vehicle, to thereby output the result fortesting of determining whether the monitored vehicle is located on saidone of the blind spots of the monitoring vehicle.

Herein, the processed values of said pieces of cue information fortesting may represent a concatenated value acquired by using said piecesof cue information for testing on the monitored vehicle generated by thecue information extracting layer 421 and the feature vector for testinggenerated by the pooling layer 43 of the detector 40. A featureconcatenation layer 440, which generates the concatenated valueincluding said pieces of cue information for testing and the featurevector for testing, may be located before the FC layer 422 formonitoring the blind spots.

Also, similarly as explained by referring to FIG. 5, the FC layer 422for monitoring the blind spots may include the neural network capable ofoutputting a result value of whether the monitored vehicle is located onsaid one of the blind spots by the multilayer perceptron to which saidpieces of cue information for testing or their processed values on themonitored vehicle are inputted.

The cue information extracting layer 421 and the FC layer 422 formonitoring the blind spots may be included in one computing device or indifferent computing devices, and also may be implemented as an algorithmperforming the operations previously mentioned, in any computingdevices.

The present invention has an effect of providing a blind spot monitoringsystem regardless of the type of a detector which detects a vehicle.

The present invention has another effect of determining whether thevehicle is located on blind spots by using an output signal from thedetector, regardless of the type of the detector.

The present invention has still another effect of minimizing cost ofmaintaining the blind spot monitoring system since the detector in theblind spot monitoring system is replaceable as needed without designinga logic.

The present invention has still yet another effect of no need to designthe logic corresponding to each detector for determining whether thevehicle is located on the blind spot, thereby minimizing time requiredto develop the blind spot monitoring system, as the blind spotmonitoring system is adaptable to all detectors.

The embodiments of the present invention as explained above can beimplemented in a form of executable program command through a variety ofcomputer means recordable to computer readable media. The computerreadable media may include solely or in combination, program commands,data files, and data structures. The program commands recorded to themedia may be components specially designed for the present invention ormay be usable to a skilled human in a field of computer software.Computer readable media include magnetic media such as hard disk, floppydisk, and magnetic tape, optical media such as CD-ROM and DVD,magneto-optical media such as floptical disk and hardware devices suchas ROM, RAM, and flash memory specially designed to store and carry outprogram commands. Program commands include not only a machine languagecode made by a compiler but also a high level language code that can beexecuted by a computer using an interpreter, etc. The hardware devicecan work as more than a software module to perform the process inaccordance with the present invention and they can do the same in theopposite case.

As seen above, the present invention has been explained by specificmatters such as detailed components, limited embodiments, and drawings.They have been provided only to help more general understanding of thepresent invention. It, however, will be understood by those skilled inthe art that various changes and modification may be made from thedescription without departing from the spirit and scope of the inventionas defined in the following claims.

Accordingly, the thought of the present invention must not be confinedto the explained embodiments, and the following patent claims as well aseverything including variations equal or equivalent to the patent claimspertain to the category of the thought of the present invention.

What is claimed is:
 1. A learning method of a CNN (Convolutional NeuralNetwork) for monitoring one or more blind spots of a monitoring vehicle,comprising steps of: (a) a learning device, if a training imagecorresponding to at least one video image taken from the monitoringvehicle is inputted, instructing a detector on the monitoring vehicle tooutput class information and location information on a monitored vehicleincluded in the training image; (b) the learning device instructing acue information extracting layer to perform one or more operations byusing the class information and the location information on themonitored vehicle, to thereby output one or more pieces of cueinformation on the monitored vehicle, and instructing an FC layer (fullyconnected layer) for monitoring the blind spots to perform one or moreneural network operations by using said pieces of cue information ortheir processed values on the monitored vehicle, to thereby output aresult of determining whether the monitored vehicle is located on one ofthe blind spots of the monitoring vehicle; and (c) the learning deviceinstructing a first loss layer to generate one or more loss values forthe blind spots by referring to the result and its corresponding firstGT (Ground Truth), to thereby learn one or more parameters of the FClayer for monitoring the blind spots by backpropagating the loss valuesfor the blind spots, and instructing a second loss layer to generate oneor more loss values for vehicle detection by referring to the classinformation and the location information on the monitored vehicle andtheir corresponding second GT, to thereby learn one or more parametersof the detector by backpropagating the loss values for the vehicledetection; wherein the detector is a vehicle detector based on an R-CNN(Region-based Convolutional Neural Network) including: one or moreconvolutional layers which generate a feature map from the trainingimage, an RPN (Region Proposal Network) which generates an ROI (RegionOf Interest) of the monitored vehicle from the feature map, a poolinglayer which generates a feature vector by pooling an area, in thefeature map, corresponding to the ROI, at least one FC layer for thevehicle detection which performs at least one fully connected operationon the feature vector, to thereby generate one or more FC output values,a classification layer which outputs the class information on themonitored vehicle by referring to the FC output values, and a regressionlayer which outputs the location information on the monitored vehicle byreferring to the FC output values.
 2. The learning method of claim 1,wherein the loss values for the vehicle detection include one or moreclass loss values and one or more location loss values of the monitoredvehicle, and wherein the learning device learns one or more parametersof the FC layer for the vehicle detection and one or more parameters ofthe convolutional layers by backpropagating the class loss values andthe location loss values.
 3. The learning method of claim 1, wherein thelearning device instructs the CNN to receive a concatenated valueacquired by using said pieces of cue information on the monitoredvehicle generated by the cue information extracting layer and thefeature vector generated by the pooling layer of the detector, as aninput to the FC layer for monitoring the blind spots.
 4. A learningmethod of a CNN (Convolutional Neural Network) for monitoring one ormore blind spots of a monitoring vehicle, comprising steps of: (a) alearning device, if a training image corresponding to at least one videoimage taken from the monitoring vehicle is inputted, instructing adetector on the monitoring vehicle to output class information andlocation information on a monitored vehicle included in the trainingimage; (b) the learning device instructing a cue information extractinglayer to perform one or more operations by using the class informationand the location information on the monitored vehicle, to thereby outputone or more pieces of cue information on the monitored vehicle, andinstructing an FC layer (fully connected layer) for monitoring the blindspots to perform one or more neural network operations by using saidpieces of cue information or their processed values on the monitoredvehicle, to thereby output a result of determining whether the monitoredvehicle is located on one of the blind spots of the monitoring vehicle;and (c) the learning device instructing a first loss layer to generateone or more loss values for the blind spots by referring to the resultand its corresponding first GT (Ground Truth), to thereby learn one ormore parameters of the FC layer for monitoring the blind spots bybackpropagating the loss values for the blind spots, and instructing asecond loss layer to generate one or more loss values for vehicledetection by referring to the class information and the locationinformation on the monitored vehicle and their corresponding second GT,to thereby learn one or more parameters of the detector bybackpropagating the loss values for the vehicle detection, wherein theFC layer for monitoring the blind spots includes a neural networkcapable of outputting a result value of whether the monitored vehicle islocated on one of the blind spots by a multilayer perceptron to whichsaid pieces of cue information on the monitored vehicle are inputted. 5.A learning method of a CNN (Convolutional Neural Network) for monitoringone or more blind spots of a monitoring vehicle, comprising steps of:(a) a learning device, if a training image corresponding to at least onevideo image taken from the monitoring vehicle is inputted, instructing adetector on the monitoring vehicle to output class information andlocation information on a monitored vehicle included in the trainingimage; (b) the learning device instructing a cue information extractinglayer to perform one or more operations by using the class informationand the location information on the monitored vehicle, to thereby outputone or more pieces of cue information on the monitored vehicle, andinstructing an FC layer (fully connected layer) for monitoring the blindspots to perform one or more neural network operations by using saidpieces of cue information or their processed values on the monitoredvehicle, to thereby output a result of determining whether the monitoredvehicle is located on one of the blind spots of the monitoring vehicle;and (c) the learning device instructing a first loss layer to generateone or more loss values for the blind spots by referring to the resultand its corresponding first GT (Ground Truth), to thereby learn one ormore parameters of the FC layer for monitoring the blind spots bybackpropagating the loss values for the blind spots, and instructing asecond loss layer to generate one or more loss values for vehicledetection by referring to the class information and the locationinformation on the monitored vehicle and their corresponding second GT,to thereby learn one or more parameters of the detector bybackpropagating the loss values for the vehicle detection, wherein saidpieces of cue information on the monitored vehicle include at least partof (i) the class information on the monitored vehicle, (ii) the locationinformation on the monitored vehicle, (iii) size information on themonitored vehicle corresponding to an ROI (Region Of Interest) size,(iv) aspect ratio information on the monitored vehicle, and (v) distanceinformation between a center of the monitored vehicle and one of outersides of the blind spots.
 6. The learning method of claim 5, wherein thecue information extracting layer determines a smaller value among eitherof (i) a distance of one outer boundary of the blind spot BS-L from thecenter of the monitored vehicle and (ii) a distance of the other outerboundary of the blind spot BS-R from the center thereof as the distanceinformation between the center of the monitored vehicle and said one ofthe outer sides of the blind spots, and further outputs relativelocation information in addition to the distance information in order todetermine whether the center of the monitored vehicle is located insideor outside of said one of the outer sides of the blind spots.
 7. Atesting method of a CNN (Convolutional Neural Network) for monitoringone or more blind spots of a monitoring vehicle, comprising steps of:(a) on condition that a learning device, (i) has instructed a detectoron the monitoring vehicle to output class information for training andlocation information for training on a monitored vehicle included in atraining image corresponding to at least one video image taken from themonitoring vehicle, (ii) has instructed a cue information extractinglayer to perform one or more operations by using the class informationfor training and the location information for training on the monitoredvehicle, to thereby output one or more pieces of cue information fortraining on the monitored vehicle, (iii) has instructed an FC layer(fully connected layer) for monitoring the blind spots to perform one ormore neural network operations by using said pieces of cue informationfor training or their processed values, to thereby output a result fortraining of determining whether the monitored vehicle is located on oneof the blind spots, and (iv) has instructed a first loss layer togenerate one or more loss values for the blind spots by referring to theresult for training and its corresponding first GT (Ground Truth), tothereby learn one or more parameters of the FC layer for monitoring theblind spots by backpropagating the loss values for the blind spots, andhas instructed a second loss layer to generate one or more loss valuesfor vehicle detection by referring to the class information for trainingand the location information for training and their corresponding secondGT, to thereby learn one or more parameters of the detector bybackpropagating the loss values for the vehicle detection, a testingdevice instructing a detector on the monitoring vehicle to output classinformation for testing and location information for testing on amonitored vehicle included in a test image taken from the monitoringvehicle; and (b) the testing device instructing the cue informationextracting layer to perform the operations by using the classinformation for testing and the location information for testing on themonitored vehicle, to thereby output one or more pieces of cueinformation for testing on the monitored vehicle, and instructing the FClayer for monitoring the blind spots to perform the neural networkoperations by using said pieces of cue information for testing or theirprocessed values on the monitored vehicle, to thereby output a resultfor testing of determining whether the monitored vehicle is located onone of the blind spots of the monitoring vehicle; wherein the detectoris a vehicle detector based on an R-CNN (Region-based ConvolutionalNeural Network) including: one or more convolutional layers whichgenerate a feature map for testing from the test image, an RPN (RegionProposal Network) which generates an ROI (Region Of Interest) fortesting of the monitored vehicle from the feature map for testing, apooling layer which generates a feature vector for testing by pooling anarea, in the feature map for testing, corresponding to the ROI fortesting, at least one FC layer for the vehicle detection which performsat least one fully connected operation on the feature vector fortesting, to thereby generate one or more FC output values, aclassification layer which outputs the class information for testing onthe monitored vehicle by referring to the FC output values, and aregression layer which outputs the location information for testing onthe monitored vehicle by referring to the FC output values.
 8. Thetesting method of claim 7, wherein the testing device instructs the CNNto receive a concatenated value for testing acquired by using saidpieces of cue information for testing on the monitored vehicle generatedby the cue information extracting layer and the feature vector fortesting generated by the pooling layer of the detector, as an input tothe FC layer for monitoring the blind spots.
 9. A testing method of aCNN (Convolutional Neural Network) for monitoring one or more blindspots of a monitoring vehicle, comprising steps of: (a) on conditionthat a learning device, (i) has instructed a detector on the monitoringvehicle to output class information for training and locationinformation for training on a monitored vehicle included in a trainingimage corresponding to at least one video image taken from themonitoring vehicle, (ii) has instructed a cue information extractinglayer to perform one or more operations by using the class informationfor training and the location information for training on the monitoredvehicle, to thereby output one or more pieces of cue information fortraining on the monitored vehicle, (iii) has instructed an FC layer(fully connected layer) for monitoring the blind spots to perform one ormore neural network operations by using said pieces of cue informationfor training or their processed values, to thereby output a result fortraining of determining whether the monitored vehicle is located on oneof the blind spots, and (iv) has instructed a first loss layer togenerate one or more loss values for the blind spots by referring to theresult for training and its corresponding first GT (Ground Truth), tothereby learn one or more parameters of the FC layer for monitoring theblind spots by backpropagating the loss values for the blind spots, andhas instructed a second loss layer to generate one or more loss valuesfor vehicle detection by referring to the class information for trainingand the location information for training and their corresponding secondGT, to thereby learn one or more parameters of the detector bybackpropagating the loss values for the vehicle detection, a testingdevice instructing a detector on the monitoring vehicle to output classinformation for testing and location information for testing on amonitored vehicle included in a test image taken from the monitoringvehicle; and (b) the testing device instructing the cue informationextracting layer to perform the operations by using the classinformation for testing and the location information for testing on themonitored vehicle, to thereby output one or more pieces of cueinformation for testing on the monitored vehicle, and instructing the FClayer for monitoring the blind spots to perform the neural networkoperations by using said pieces of cue information for testing or theirprocessed values on the monitored vehicle, to thereby output a resultfor testing of determining whether the monitored vehicle is located onone of the blind spots of the monitoring vehicle; wherein the FC layerfor monitoring the blind spots includes a neural network capable ofoutputting a result value for testing of whether the monitored vehicleis located on one of the blind spots by a multilayer perceptron to whichsaid pieces of cue information for testing or their processed values onthe monitored vehicle are inputted.
 10. A testing method of a CNN(Convolutional Neural Network) for monitoring one or more blind spots ofa monitoring vehicle, comprising steps of: (a) on condition that alearning device, (i) has instructed a detector on the monitoring vehicleto output class information for training and location information fortraining on a monitored vehicle included in a training imagecorresponding to at least one video image taken from the monitoringvehicle, (ii) has instructed a cue information extracting layer toperform one or more operations by using the class information fortraining and the location information for training on the monitoredvehicle, to thereby output one or more pieces of cue information fortraining on the monitored vehicle, (iii) has instructed an FC layer(fully connected layer) for monitoring the blind spots to perform one ormore neural network operations by using said pieces of cue informationfor training or their processed values, to thereby output a result fortraining of determining whether the monitored vehicle is located on oneof the blind spots, and (iv) has instructed a first loss layer togenerate one or more loss values for the blind spots by referring to theresult for training and its corresponding first GT (Ground Truth), tothereby learn one or more parameters of the FC layer for monitoring theblind spots by backpropagating the loss values for the blind spots, andhas instructed a second loss layer to generate one or more loss valuesfor vehicle detection by referring to the class information for trainingand the location information for training and their corresponding secondGT, to thereby learn one or more parameters of the detector bybackpropagating the loss values for the vehicle detection, a testingdevice instructing a detector on the monitoring vehicle to output classinformation for testing and location information for testing on amonitored vehicle included in a test image taken from the monitoringvehicle; and (b) the testing device instructing the cue informationextracting layer to perform the operations by using the classinformation for testing and the location information for testing on themonitored vehicle, to thereby output one or more pieces of cueinformation for testing on the monitored vehicle, and instructing the FClayer for monitoring the blind spots to perform the neural networkoperations by using said pieces of cue information for testing or theirprocessed values on the monitored vehicle, to thereby output a resultfor testing of determining whether the monitored vehicle is located onone of the blind spots of the monitoring vehicle; wherein said pieces ofcue information for testing on the monitored vehicle include at leastpart of (i) the class information for testing on the monitored vehicle,(ii) the location information for testing on the monitored vehicle,(iii) size information for testing on the monitored vehiclecorresponding to an ROI size, (iv) aspect ratio information for testingon the monitored vehicle, and (v) distance information for testingbetween a center of the monitored vehicle and one of outer sides of theblind spots.
 11. The testing method of claim 10, wherein the cueinformation extracting layer determines a smaller value among either of(i) a distance of one outer boundary of the blind spot BS-L from thecenter of the monitored vehicle and (ii) a distance of the other outerboundary of the blind spot BS-R from the center thereof as the distanceinformation for testing between the center of the monitored vehicle andsaid one of the outer sides of the blind spots, and further outputsrelative location information for testing in addition to the distanceinformation for testing in order to determine whether the center of themonitored vehicle is located inside or outside of said one of the outersides of the blind spots.
 12. A testing method of a CNN (ConvolutionalNeural Network) for monitoring one or more blind spots of a monitoringvehicle, comprising steps of: (a) on condition that a learning device,(i) has instructed a detector on the monitoring vehicle to output classinformation for training and location information for training on amonitored vehicle included in a training image corresponding to at leastone video image taken from the monitoring vehicle, (ii) has instructed acue information extracting layer to perform one or more operations byusing the class information for training and the location informationfor training on the monitored vehicle, to thereby output one or morepieces of cue information for training on the monitored vehicle, (iii)has instructed an FC layer (fully connected layer) for monitoring theblind spots to perform one or more neural network operations by usingsaid pieces of cue information for training or their processed values,to thereby output a result for training of determining whether themonitored vehicle is located on one of the blind spots, and (iv) hasinstructed a first loss layer to generate one or more loss values forthe blind spots by referring to the result for training and itscorresponding first GT (Ground Truth), to thereby learn one or moreparameters of the FC layer for monitoring the blind spots bybackpropagating the loss values for the blind spots, and has instructeda second loss layer to generate one or more loss values for vehicledetection by referring to the class information for training and thelocation information for training and their corresponding second GT, tothereby learn one or more parameters of the detector by backpropagatingthe loss values for the vehicle detection, a testing device instructinga detector on the monitoring vehicle to output class information fortesting and location information for testing on a monitored vehicleincluded in a test image taken from the monitoring vehicle; and (b) thetesting device instructing the cue information extracting layer toperform the operations by using the class information for testing andthe location information for testing on the monitored vehicle, tothereby output one or more pieces of cue information for testing on themonitored vehicle, and instructing the FC layer for monitoring the blindspots to perform the neural network operations by using said pieces ofcue information for testing or their processed values on the monitoredvehicle, to thereby output a result for testing of determining whetherthe monitored vehicle is located on one of the blind spots of themonitoring vehicle, wherein the monitoring vehicle for the testingdevice is not identical to the monitoring vehicle for the learningdevice.
 13. A learning device of a CNN (Convolutional Neural Network)for monitoring one or more blind spots of a monitoring vehicle,comprising: a communication part for receiving a training imagecorresponding to at least one video image taken from the monitoringvehicle; and a processor for (I) instructing a detector on themonitoring vehicle to output class information and location informationon a monitored vehicle included in the training image, (II) instructinga cue information extracting layer to perform one or more operations byusing the class information and the location information on themonitored vehicle, to thereby output one or more pieces of cueinformation on the monitored vehicle, and instructing an FC layer (fullyconnected layer) for monitoring the blind spots to perform one or moreneural network operations by using said pieces of cue information ortheir processed values on the monitored vehicle, to thereby output aresult of determining whether the monitored vehicle is located on one ofthe blind spots of the monitoring vehicle, and (III) instructing a firstloss layer to generate one or more loss values for the blind spots byreferring to the result and its corresponding first GT (Ground Truth),to thereby learn one or more parameters of the FC layer for monitoringthe blind spots by backpropagating the loss values for the blind spots,and instructing a second loss layer to generate one or more loss valuesfor vehicle detection by referring to the class information and thelocation information on the monitored vehicle and their correspondingsecond GT, to thereby learn one or more parameters of the detector bybackpropagating the loss values for the vehicle detection, wherein thedetector is a vehicle detector based on an R-CNN (Region-basedConvolutional Neural Network) including: one or more convolutionallayers which generate a feature map from the training image, an RPN(Region Proposal Network) which generates an ROI (Region Of Interest) ofthe monitored vehicle from the feature map, a pooling layer whichgenerates a feature vector by pooling an area, in the feature map,corresponding to the ROI, at least one FC layer for the vehicledetection which performs at least one fully connected operation on thefeature vector, to thereby generate one or more FC output values, aclassification layer which outputs the class information on themonitored vehicle by referring to the FC output values, and a regressionlayer which outputs the location information on the monitored vehicle byreferring to the FC output values.
 14. The learning device of claim 13,wherein the loss values for the vehicle detection include one or moreclass loss values and one or more location loss values of the monitoredvehicle, and wherein the processor learns one or more parameters of theFC layer for the vehicle detection and one or more parameters of theconvolutional layers by backpropagating the class loss values and thelocation loss values.
 15. The learning device of claim 13, wherein theprocessor instructs the CNN to receive a concatenated value acquired byusing said pieces of cue information on the monitored vehicle generatedby the cue information extracting layer and the feature vector generatedby the pooling layer of the detector, as an input to the FC layer formonitoring the blind spots.
 16. A learning device of a CNN(Convolutional Neural Network) for monitoring one or more blind spots ofa monitoring vehicle, comprising: a communication part for receiving atraining image corresponding to at least one video image taken from themonitoring vehicle; and a processor for (I) instructing a detector onthe monitoring vehicle to output class information and locationinformation on a monitored vehicle included in the training image, (II)instructing a cue information extracting layer to perform one or moreoperations by using the class information and the location informationon the monitored vehicle, to thereby output one or more pieces of cueinformation on the monitored vehicle, and instructing an FC layer (fullyconnected layer) for monitoring the blind spots to perform one or moreneural network operations by using said pieces of cue information ortheir processed values on the monitored vehicle, to thereby output aresult of determining whether the monitored vehicle is located on one ofthe blind spots of the monitoring vehicle, and (III) instructing a firstloss layer to generate one or more loss values for the blind spots byreferring to the result and its corresponding first GT (Ground Truth),to thereby learn one or more parameters of the FC layer for monitoringthe blind spots by backpropagating the loss values for the blind spots,and instructing a second loss layer to generate one or more loss valuesfor vehicle detection by referring to the class information and thelocation information on the monitored vehicle and their correspondingsecond GT, to thereby learn one or more parameters of the detector bybackpropagating the loss values for the vehicle detection, wherein theFC layer for monitoring the blind spots includes a neural networkcapable of outputting a result value of whether the monitored vehicle islocated on one of the blind spots by a multilayer perceptron to whichsaid pieces of cue information on the monitored vehicle are inputted.17. A learning device of a CNN (Convolutional Neural Network) formonitoring one or more blind spots of a monitoring vehicle, comprising:a communication part for receiving a training image corresponding to atleast one video image taken from the monitoring vehicle; and a processorfor (I) instructing a detector on the monitoring vehicle to output classinformation and location information on a monitored vehicle included inthe training image, (II) instructing a cue information extracting layerto perform one or more operations by using the class information and thelocation information on the monitored vehicle, to thereby output one ormore pieces of cue information on the monitored vehicle, and instructingan FC layer (fully connected layer) for monitoring the blind spots toperform one or more neural network operations by using said pieces ofcue information or their processed values on the monitored vehicle, tothereby output a result of determining whether the monitored vehicle islocated on one of the blind spots of the monitoring vehicle, and (III)instructing a first loss layer to generate one or more loss values forthe blind spots by referring to the result and its corresponding firstGT (Ground Truth), to thereby learn one or more parameters of the FClayer for monitoring the blind spots by backpropagating the loss valuesfor the blind spots, and instructing a second loss layer to generate oneor more loss values for vehicle detection by referring to the classinformation and the location information on the monitored vehicle andtheir corresponding second GT, to thereby learn one or more parametersof the detector by backpropagating the loss values for the vehicledetection, wherein said pieces of cue information on the monitoredvehicle include at least part of (i) the class information on themonitored vehicle, (ii) the location information on the monitoredvehicle, (iii) size information on the monitored vehicle correspondingto an ROI (Region Of Interest) size, (iv) aspect ratio information onthe monitored vehicle, and (v) distance information between a center ofthe monitored vehicle and one of outer sides of the blind spots.
 18. Thelearning device of claim 17, wherein the cue information extractinglayer determines a smaller value among either of (i) a distance of oneouter boundary of the blind spot BS-L from the center of the monitoredvehicle and (ii) a distance of the other outer boundary of the blindspot BS-R from the center thereof as the distance information betweenthe center of the monitored vehicle and said one of the outer sides ofthe blind spots, and further outputs relative location information inaddition to the distance information in order to determine whether thecenter of the monitored vehicle is located inside or outside of said oneof the outer sides of the blind spots.
 19. A testing device of a CNN(Convolutional Neural Network) for monitoring one or more blind spots ofa monitoring vehicle, comprising: a communication part for, on conditionthat a learning device, (i) has instructed a detector on the monitoringvehicle to output class information for training and locationinformation for training on a monitored vehicle included in a trainingimage corresponding to at least one video image taken from themonitoring vehicle, (ii) has instructed a cue information extractinglayer to perform one or more operations by using the class informationfor training and the location information for training on the monitoredvehicle, to thereby output one or more pieces of cue information fortraining on the monitored vehicle, (iii) has instructed an FC layer(fully connected layer) for monitoring the blind spots to perform one ormore neural network operations by using said pieces of cue informationfor training or their processed values, to thereby output a result fortraining of determining whether the monitored vehicle is located on oneof the blind spots, and (iv) has instructed a first loss layer togenerate one or more loss values for the blind spots by referring to theresult for training and its corresponding first GT (Ground Truth), tothereby learn one or more parameters of the FC layer for monitoring theblind spots by backpropagating the loss values for the blind spots, andhas instructed a second loss layer to generate one or more loss valuesfor vehicle detection by referring to the class information for trainingand the location information for training and their corresponding secondGT, to thereby learn one or more parameters of the detector bybackpropagating the loss values for the vehicle detection, acquiringclass information for testing and location information for testing onthe monitored vehicle form the detector which detects the monitoredvehicle in a test image taken from the monitoring vehicle; and aprocessor for (I) instructing the cue information extracting layer toperform the operations by using the class information for testing andthe location information for testing on the monitored vehicle, tothereby output one or more pieces of cue information for testing on themonitored vehicle, and (II) instructing the FC layer for monitoring theblind spots to perform the neural network operations by using saidpieces of cue information for testing or their processed values on themonitored vehicle, to thereby output a result for testing of determiningwhether the monitored vehicle is located on one of the blind spots ofthe monitoring vehicle, wherein the detector is a vehicle detector basedon an R-CNN (Region-based Convolutional Neural Network) including: oneor more convolutional layers which generate a feature map for testingfrom the test image, an RPN (Region Proposal Network) which generates anROI (Region Of Interest) for testing of the monitored vehicle from thefeature map for testing, a pooling layer which generates a featurevector for testing by pooling an area, in the feature map for testing,corresponding to the ROI for testing, at least one FC layer for thevehicle detection which performs at least one fully connected operationon the feature vector for testing, to thereby generate one or more FCoutput values, a classification layer which outputs the classinformation for testing on the monitored vehicle by referring to the FCoutput values, and a regression layer which outputs the locationinformation for testing on the monitored vehicle by referring to the FCoutput values.
 20. The testing device of claim 19, wherein the processorinstructs the CNN to receive a concatenated value for testing acquiredby using said pieces of cue information for testing on the monitoredvehicle generated by the cue information extracting layer and thefeature vector for testing generated by the pooling layer of thedetector, as an input to the FC layer for monitoring the blind spots.21. A testing device of a CNN (Convolutional Neural Network) formonitoring one or more blind spots of a monitoring vehicle, comprising:a communication part for, on condition that a learning device, (i) hasinstructed a detector on the monitoring vehicle to output classinformation for training and location information for training on amonitored vehicle included in a training image corresponding to at leastone video image taken from the monitoring vehicle, (ii) has instructed acue information extracting layer to perform one or more operations byusing the class information for training and the location informationfor training on the monitored vehicle, to thereby output one or morepieces of cue information for training on the monitored vehicle, (iii)has instructed an FC layer (fully connected layer) for monitoring theblind spots to perform one or more neural network operations by usingsaid pieces of cue information for training or their processed values,to thereby output a result for training of determining whether themonitored vehicle is located on one of the blind spots, and (iv) hasinstructed a first loss layer to generate one or more loss values forthe blind spots by referring to the result for training and itscorresponding first GT (Ground Truth), to thereby learn one or moreparameters of the FC layer for monitoring the blind spots bybackpropagating the loss values for the blind spots, and has instructeda second loss layer to generate one or more loss values for vehicledetection by referring to the class information for training and thelocation information for training and their corresponding second GT, tothereby learn one or more parameters of the detector by backpropagatingthe loss values for the vehicle detection, acquiring class informationfor testing and location information for testing on the monitoredvehicle form the detector which detects the monitored vehicle in a testimage taken from the monitoring vehicle; and a processor for (I)instructing the cue information extracting layer to perform theoperations by using the class information for testing and the locationinformation for testing on the monitored vehicle, to thereby output oneor more pieces of cue information for testing on the monitored vehicle,and (II) instructing the FC layer for monitoring the blind spots toperform the neural network operations by using said pieces of cueinformation for testing or their processed values on the monitoredvehicle, to thereby output a result for testing of determining whetherthe monitored vehicle is located on one of the blind spots of themonitoring vehicle, wherein the FC layer for monitoring the blind spotsincludes a neural network capable of outputting a result value fortesting of whether the monitored vehicle is located on one of the blindspots by a multilayer perceptron to which said pieces of cue informationfor testing or their processed values on the monitored vehicle areinputted.
 22. A testing device of a CNN (Convolutional Neural Network)for monitoring one or more blind spots of a monitoring vehicle,comprising: a communication part for, on condition that a learningdevice, (i) has instructed a detector on the monitoring vehicle tooutput class information for training and location information fortraining on a monitored vehicle included in a training imagecorresponding to at least one video image taken from the monitoringvehicle, (ii) has instructed a cue information extracting layer toperform one or more operations by using the class information fortraining and the location information for training on the monitoredvehicle, to thereby output one or more pieces of cue information fortraining on the monitored vehicle, (iii) has instructed an FC layer(fully connected layer) for monitoring the blind spots to perform one ormore neural network operations by using said pieces of cue informationfor training or their processed values, to thereby output a result fortraining of determining whether the monitored vehicle is located on oneof the blind spots, and (iv) has instructed a first loss layer togenerate one or more loss values for the blind spots by referring to theresult for training and its corresponding first GT (Ground Truth), tothereby learn one or more parameters of the FC layer for monitoring theblind spots by backpropagating the loss values for the blind spots, andhas instructed a second loss layer to generate one or more loss valuesfor vehicle detection by referring to the class information for trainingand the location information for training and their corresponding secondGT, to thereby learn one or more parameters of the detector bybackpropagating the loss values for the vehicle detection, acquiringclass information for testing and location information for testing onthe monitored vehicle form the detector which detects the monitoredvehicle in a test image taken from the monitoring vehicle; and aprocessor for (I) instructing the cue information extracting layer toperform the operations by using the class information for testing andthe location information for testing on the monitored vehicle, tothereby output one or more pieces of cue information for testing on themonitored vehicle, and (II) instructing the FC layer for monitoring theblind spots to perform the neural network operations by using saidpieces of cue information for testing or their processed values on themonitored vehicle, to thereby output a result for testing of determiningwhether the monitored vehicle is located on one of the blind spots ofthe monitoring vehicle, wherein said pieces of cue information fortesting on the monitored vehicle include at least part of (i) the classinformation for testing on the monitored vehicle, (ii) the locationinformation for testing on the monitored vehicle, (iii) size informationfor testing on the monitored vehicle corresponding to an ROI size, (iv)aspect ratio information for testing on the monitored vehicle, and (v)distance information for testing between a center of the monitoredvehicle and one of outer sides of the blind spots.
 23. The testingdevice of claim 22, wherein the cue information extracting layerdetermines a smaller value among either of (i) a distance of one outerboundary of the blind spot BS-L from the center of the monitored vehicleand (ii) a distance of the other outer boundary of the blind spot BS-Rfrom the center thereof as the distance information for testing betweenthe center of the monitored vehicle and said one of the outer sides ofthe blind spots, and further outputs relative location information fortesting in addition to the distance information for testing in order todetermine whether the center of the monitored vehicle is located insideor outside of said one of the outer sides of the blind spots.
 24. Atesting device of a CNN (Convolutional Neural Network) for monitoringone or more blind spots of a monitoring vehicle, comprising: acommunication part for, on condition that a learning device, (i) hasinstructed a detector on the monitoring vehicle to output classinformation for training and location information for training on amonitored vehicle included in a training image corresponding to at leastone video image taken from the monitoring vehicle, (ii) has instructed acue information extracting layer to perform one or more operations byusing the class information for training and the location informationfor training on the monitored vehicle, to thereby output one or morepieces of cue information for training on the monitored vehicle, (iii)has instructed an FC layer (fully connected layer) for monitoring theblind spots to perform one or more neural network operations by usingsaid pieces of cue information for training or their processed values,to thereby output a result for training of determining whether themonitored vehicle is located on one of the blind spots, and (iv) hasinstructed a first loss layer to generate one or more loss values forthe blind spots by referring to the result for training and itscorresponding first GT (Ground Truth), to thereby learn one or moreparameters of the FC layer for monitoring the blind spots bybackpropagating the loss values for the blind spots, and has instructeda second loss layer to generate one or more loss values for vehicledetection by referring to the class information for training and thelocation information for training and their corresponding second GT, tothereby learn one or more parameters of the detector by backpropagatingthe loss values for the vehicle detection, acquiring class informationfor testing and location information for testing on the monitoredvehicle form the detector which detects the monitored vehicle in a testimage taken from the monitoring vehicle; and a processor for (I)instructing the cue information extracting layer to perform theoperations by using the class information for testing and the locationinformation for testing on the monitored vehicle, to thereby output oneor more pieces of cue information for testing on the monitored vehicle,and (II) instructing the FC layer for monitoring the blind spots toperform the neural network operations by using said pieces of cueinformation for testing or their processed values on the monitoredvehicle, to thereby output a result for testing of determining whetherthe monitored vehicle is located on one of the blind spots of themonitoring vehicle, wherein the monitoring vehicle for the testingdevice is not identical to the monitoring vehicle for the learningdevice.